Lexical semantic content, not syntactic structure, is the main contributor to ANN-brain similarity of fMRI responses in the language network

Abstract Representations from artificial neural network (ANN) language models have been shown to predict human brain activity in the network. To understand what aspects of linguistic stimuli contribute ANN-to-brain similarity, we used an fMRI dataset responses n = 627 naturalistic English sentences (Pereira et al., 2018) and systematically manipulated for which ANN representations were extracted. In particular, i) perturbed sentences’ word order, ii) removed different subsets words, or iii) replaced with other varying semantic similarity. We found that lexical content sentence (largely carried by words) rather than sentence’s syntactic form (conveyed via order function is primarily responsible follow-up analyses, perturbation manipulations adversely affect predictivity also lead more divergent ANN’s embedding space decrease ability upcoming tokens those stimuli. Further, results are robust whether mapping model trained on intact stimuli, conditioned same context humans saw. The critical result—that lexical-semantic main contributor similarity between ones—aligns idea goal system extract meaning strings. Finally, this work highlights strength systematic experimental evaluating how close accurate generalizable